feat(observability): record event loop stalls in the server trace - #13697

A user on T3 Code 0.0.40 saw the server get slow after 9 days of uptime (about 5,000 threads, 130 projects, about 30 Claude provider instances). A 5 s timer fired 10-14 s late, so the event loop likely stalled for many seconds. But nothing in the server measures event loop delay, so this was a guess, not a measurement. Metrics and logs outside spans do not reach disk without OTLP, so most users would have no record either way.

How I fixed it

• New apps/server/src/observability/EventLoopMonitor.ts, a scoped layer next to the app observability layer in server.ts.
• It turns on Node's monitorEventLoopDelay with a 1 s resolution. Every 30 s it reads the max delay, event loop utilization and active time, the process.resourceUsage() delta (user and system CPU, major and minor page faults, involuntary context switches), and RSS. Then it resets the histogram.
• When the max delay is over 2 s, it writes a root server.eventLoop.stall span at Warn trace level, with a warning log and those numbers as attributes. The span goes to server.trace.ndjson, and the warning shows in Settings > Diagnostics.
• The histogram records the full gap between its timer wakeups, so an idle loop reads about 1 s. The code subtracts the resolution, so every stall over 3 s is recorded. That still catches the reported 10-14 s stalls.
• No false warnings from system sleep: libuv's clock keeps running while macOS and Windows sleep, so a wake reads as a long delay. A sample only counts when event loop active time is at least the delay. A real stall is active time; a sleep is idle time in poll.
• No warning at every launch: the monitor builds before the rest of the server, so the first sample covers startup work such as migrations. It is discarded.
• Cost: 1 native timer wakeup per second that never enters JS, one small JS tick every 30 s, and at most one span per 30 s during a stall storm.
• The sampler fiber is forked at layer build, so it keeps no parent span (the same concern as #9824).
• docs/operations/observability.md: new "Event Loop Stalls" section after "Metrics". It says what the span means, what is not recorded, and how to read CPU and page faults against the stall. High system CPU with many faults means memory pressure (on macOS, compressed memory shows as minor faults plus system CPU).

Verification

• vp test run src/observability/EventLoopMonitor.test.ts (in apps/server):
  • A layer test with a stub sampler and TestClock. A stalled first sample is discarded, a sample under the threshold (after the resolution is subtracted) records nothing, and the next stalled sample records one span with the expected attributes and a warning.
  • A test of the decision function: a 10 minute delay with little active time (a system sleep) is not a stall, and the same delay with enough active time is.
  • Each test fails if its fix is removed.
• vp lint and vp fmt on the changed files.
• vp run --filter t3 typecheck.

Made by Claude Opus 5.5 (1M context) in Claude Code, running in T3 Code.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added server event-loop stall monitoring that records warning spans with timing and resource-usage details for stalls over 2 seconds when the server is sufficiently active. The first sample after startup is skipped.
• Documentation
  • Updated guidance on the 2-second threshold, potential undercounting of shorter stalls, sample-boundary limitations, and how sleep time is measured on macOS and Windows. Added notes on identifying possible false stall reports.